Synthetic representation of a surgical robot

ABSTRACT

A system comprises a first robotic arm adapted to support and move a tool and a second robotic arm adapted to support and move a camera. The system also comprises an input device, a display, and a processor. The processor is configured to, in a first mode, command the first robotic arm to move the camera in response to a first input received from the input device to capture an image of the tool and present the image as a displayed image on the display. The processor is configured to, in a second mode, display a synthetic image of the first robotic arm in a boundary area around the captured image on the display, and in response to a second input, change a size of the boundary area relative a size of the displayed image.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation in part of U.S. patent applicationSer. No. 11/478,531 (filed Jun. 29, 2006) and a continuation in part ofU.S. patent application Ser. No. 12/163,087 (filed Jun. 27, 2008), bothof which are incorporated herein by reference.

BACKGROUND

Minimally invasive surgeries performed by robotic surgical systems areknown and commonly used in remote or in other environments where it isadvantageous for a human not to perform surgery. One example of such atelerobotic surgical system is the minimally invasive robotic surgerysystem described in commonly owned U.S. Pat. No. 7,155,315. The daVinci® Surgical Systems manufactured by Intuitive Surgical, Inc. ofSunnyvale, Calif. are illustrative implementations of minimally invasiverobotic surgical systems (e.g., teleoperated; telesurgical).

A common form of minimally invasive surgery is endoscopy. Endoscopicsurgical instruments in minimally invasive medical techniques generallyinclude an endoscope for viewing the surgical field, and working toolsthat include end effectors. Typical surgical end effectors includeclamps, graspers, scissors, staplers, or needle holders, as examples.The working tools are similar to those used in conventional (open)surgery, except that the end effector of each tool is supported on theend of, for example, an approximately 12-inch-long extension tube.

To manipulate end effectors, a human operator, typically a surgeon,manipulates or otherwise commands a locally-provided master manipulator.Commands from the master manipulator are translated as appropriate andsent to a remotely-deployed slave manipulator. The slave manipulatorthen manipulates the end effectors according to the operator's commands.

Force feedback may be included in minimally invasive robotic surgicalsystems. To provide such feedback, the remote slave manipulatorstypically provide force information to the master manipulator, and thatforce information is utilized to provide force feedback to the surgeonso that the surgeon is given the perception of feeling forces acting ona slave manipulator. In some force feedback implementations, hapticfeedback may provide an artificial feel to the surgeon of tissuereactive forces on a working tool and its end effector.

Often, the master controls, which are typically located at a surgeonconsole, will include a clutch or other device for releasing one of thework tools at the patient site. This feature may be used, for example,in a system where there are more than two working tools. In such asystem, the surgeon may release control of one working tool by onemaster and establish control over another working tool with that master.

The surgeon typically views an image of only the distal ends of theworking tools that are within the endoscope's field of view. The surgeoncannot see portions of a tool, or an entire tool, that is outside thefield of view. Accordingly, the surgeon cannot see if two or more toolsare interfering with each other outside the field of view. Further,since the endoscope may be manipulated to be at various positions andorientations with reference to a surgical site and to the surgeon's bodyframe of reference, the surgeon may become confused about the generallocation of the tools. Consequently, the surgeon may not understand howto best move the master manipulators to avoid an inter-tool interferenceor to reorient one or more tools with reference to the surgical site.

SUMMARY

The following presents a simplified summary of some aspects andembodiments of the invention in order to provide a basic understandingof the invention. This summary is not an extensive overview of theinvention. It is not intended to identify key/critical elements of theinvention or to delineate the scope of the invention. Its sole purposeis to present some aspects and embodiments of the invention in asimplified form as a prelude to the more detailed description that ispresented later.

In an embodiment, a robotic surgical system is provided. The systemincludes a robot including a linkage supporting at least one tool forperforming surgery on a patient; a kinematic component coupled to therobot so as to obtain joint state information from the linkage; adisplay; and a first component coupling the display with the kinematiccomponent so as to display a synthetic representation of the robotincluding a graphical representation of at least a portion of thelinkage based upon linkage structure data regarding the linkage; and thejoint state information.

In another embodiment, a robotic surgical system is provided. The systemincludes a robot including an image capture device having a field ofview and a linkage supporting at least one tool for performing surgeryon a patient; a kinematic component coupled to the linkage so as toobtain joint states information regarding the linkage; data regardingstructure of the first linkage and said at least one tool; and acollision detection component coupled to the data and to the kinematiccomponent so as to generate a warning.

In still another embodiment, a method of controlling a position of atool in a robotic system is provided. The method includes displaying afirst image on a display, the first image comprising a video feed of atool or linkage of a robot within a field of view; displaying a secondimage on the display, the second image representing a three dimensionalmodel of the tool or linkage, with the second image of the threedimensional model aligned with first image of the tool or linkage; andmoving an input device with reference to the first and second images onthe display so as to control movement of the tool or linkage.

In yet still another embodiment, a method of providing a range of motionof a tool of a robotic system is provided. The method includesdisplaying a first image representing a position of the tool; andsuperimposing on the first image a second image representing a limit ofmotion of the tool.

In yet another embodiment, a robotic system is provided. The methodincludes maintaining information about a position of a tool of a roboticsystem; and generating a signal as a result of the tool being within athreshold distance from a limit of motion of the tool.

In another embodiment, a robotic surgical system is provided. The systemincludes a robot including a linkage supporting at least one tool forperforming surgery on a patient; an image capture device having a fieldof view encompassing the tool; a kinematic component coupled to therobot so as to obtain joint state information from the linkage; adisplay coupled to the image capture device to display the field ofview; and a first component coupling the display with the kinematiccomponent so as to display information on the tool represented in thefield of view, the position of the information being based upon linkagestructure data regarding the linkage; and the joint state information.

In still another embodiment, a method in a robotic system is provided.The method includes displaying a first image comprising a video feed ofa tool supported by a robot within a field of view; and displaying asynthetic three-dimensional representation of the robot including thetool.

In another embodiment, a method in a robotic system is provided. Themethod includes displaying a first image comprising a video feed of atool supported by a robot within a field of view, the first imageconsisting of a first portion of the robot; and displaying a syntheticthree-dimensional representation of the robot including the tool, withthe synthetic three-dimensional representation comprising a secondportion of the robot that is greater than the first portion.

In yet another embodiment, a method is provided in a robotic system, themethod including displaying a first image comprising a video feed of atool supported by a robot within a field of view, the first imageconsisting of a first portion of the robot viewed from a firstdirection; and displaying a synthetic three-dimensional representationof the robot including the tool, with the synthetic three-dimensionalrepresentation viewed from a second direction.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a top view of an operating room which includes a minimallyinvasive telesurgical system;

FIG. 2 is front view of a patient cart for the minimally invasivetelesurgical system of FIG. 1;

FIG. 3 is a block diagram representing components of the minimallyinvasive telesurgical system of FIG. 1;

FIG. 4 is a block diagram representing components for a computer for usein the minimally invasive telesurgical system of FIG. 1;

FIG. 5 is a side perspective view of a master controller;

FIG. 6 is a view of a synthetic image of a robot;

FIG. 7 is a flowchart representing a process for updating a rendering ofa synthetic image;

FIG. 8 is a view provided by a display that provides both a field ofview for an endoscope and a synthetic image of a robot supporting theendoscope;

FIG. 9 shows a tile window displaying an alternate angle for viewing aportion of the synthetic image of a robot;

FIG. 10 shows a field of view in which two tools are colliding;

FIG. 11 is a flowchart showing a process for providing collisioninformation;

FIG. 12 is a flow chart representing a process for lost tool recovery;

FIG. 13 shows a field of view projected over a window tile that includesa synthetic image of a robot; and

FIG. 14 is a flow chart representing a process for displayinginformation utilizing a modeling component.

DETAILED DESCRIPTION

In the following description, various aspects and embodiments of thepresent invention will be described. For purposes of explanation,specific configurations and details are set forth in order to provide athorough understanding of the embodiments. However, it will also beapparent to one skilled in the art that the present invention may bepracticed without the specific details. Furthermore, well-known featuresmay be omitted from this description or simplified in order not toobscure the embodiment being described.

Referring now to the drawings, in which like reference numeralsrepresent like parts throughout several views, FIG. 1 shows a minimallyinvasive telesurgical system 20 having an operator station or surgeonconsole 30 in accordance with an embodiment. The surgeon console 30includes a viewer 32 where an image of a surgical site is displayed to asurgeon S. As is known, a support (not shown) is provided on which thesurgeon S can rest his or her forearms while gripping two mastercontrols 700 (FIG. 5), one in each hand. More controls may be providedif more end effectors are available, but typically a surgeon manipulatesonly two controls at a time and, if multiple tools are used, the surgeonreleases one tool with a master control 700 and grasps another with samemaster control. When using the surgeon console 30, the surgeon Stypically sits in a chair in front of the surgeon console, positions hisor her eyes in front of the viewer 32, and grips the master controls700, one in each hand, while resting his or her forearms on the support.

A patient side cart 40 of the telesurgical system 20 is positionedadjacent to a patient P. In use, the patient side cart 40 is positionedclose to the patient P requiring surgery. The patient side cart 40typically is stationary during a surgical procedure, and includes wheelsor castors to render it mobile. The surgeon console 30 is typicallypositioned remote from the patient side cart 40, and it may be separatedfrom the patient side cart by a great distance—even miles away—but willtypically be used within the same operating room as the patient sidecart.

The patient side cart 40, shown in more detail in FIG. 2, typicallyincludes two or more robotic arm assemblies. In the embodiment shown inFIG. 2, the patient side cart 40 includes four robotic arm assemblies42, 44, 46, 48, but more or less may be provided. Each robotic armassembly 42, 44, 46, 48 is normally operatively connected to one of themaster controls of the surgeon console 30. Thus, movement of themanipulator portion of the robotic arm assemblies 44, 46 48 iscontrolled by manipulation of the master controls.

One of the robotic arm assemblies, indicated by the reference numeral42, is arranged to hold an image capture device 50, e.g., an endoscope,or the like. The endoscope or image capture device 50 includes a viewingend 56 at a remote end of an elongated shaft 54. The elongated shaft 54permits the viewing end 56 to be inserted through a surgery entry portof the patient P. The image capture device 50 is operatively connectedto the viewer 32 of the surgeon console 30 to display an image capturedat its viewing end 56.

Each of the other robotic arm assemblies 44, 46, 48 is a linkage thatsupports and includes a removable surgical instrument or tool 60, 62,64, respectively. The tools 60, 62, 64 of the robotic arm assemblies 44,46, 48 include end effectors 66, 68, 70, respectively. The end effectors66, 68, 70 are mounted on wrist members which are mounted on distal endsof elongated shafts of the tools, as is known in the art. The tools 60,62, 64 have elongated shafts to permit the end effectors 66, 68, 70 tobe inserted through surgical entry ports of the patient P. Movement ofthe end effectors 66, 68, 70 relative to the ends of the shafts of thetools 60, 62, 64 is controlled by the master controls of the surgeonconsole 30.

The depicted telesurgical system 20 includes a vision cart 80, whichcontains equipment associated with the image capture device. In anotherembodiment, the vision cart 80 can be combined with other equipment thatincludes most of the computer equipment or other controls (the “core”data processing equipment) for operating the telesurgical system 20. Asan example, signals sent by the master controllers of the surgeonconsole 30 may be sent to the vision/core cart 80, which in turn mayinterpret the signals and generate commands for the end effectors 66,68, 70 and/or robotic arm assemblies 44, 46, 48. In addition, video sentfrom the image capture device 50 to the viewer 34 may be processed by,or simply transferred by, the vision cart 80.

FIG. 3 is a diagrammatic representation of the telesurgical system 20.As can be seen, the system includes the surgeon console 30, the patientside cart 40, and the vision cart 80. In addition, in accordance with anembodiment, an additional computer 82 and display 84 are provided. Thesecomponents may be incorporated in one or more of the surgeon console 30,the patient side cart 40, and/or the vision cart 80. For example, thefeatures of the computer 82 may be incorporated into the vision cart 80.In addition, the features of the display 84 may be incorporated into thesurgeon console 30, for example, in the viewer 32, or maybe provided bya completely separate display at the surgeon console or on anotherlocation. In addition, in accordance with an embodiment, the computer 82may generate information that may be utilized without a display, such asthe display 84.

Although described as a “computer,” the computer 82 may be a componentof a computer system or any other software or hardware that is capableof performing the functions described herein. Moreover, as describedabove, functions and features of the computer 82 may be distributed overseveral devices or software components. Thus, the computer 82 shown inthe drawings is for the convenience of discussion, and it may bereplaced by a controller or its functions may be provided by one or moreother components.

FIG. 4 shows components of the computer 82 in accordance with anembodiment. A positional component is included in or is otherwiseassociated with the computer 82. The positional component providesinformation about a position of an end effector, such as one of the endeffectors 66, 68, 70. In the embodiment shown in the drawings, a tooltracking component 90 is used for the positional component and providesinformation about a position of an end effector, such as the endeffectors 66, 68, 70. As used herein, “position” means at least one ofthe location and/or the orientation of the end effector. A variety ofdifferent technologies may be used to provide information about aposition of an end effector, and such technologies may or may not beconsidered tool tracking devices. In a simple embodiment, the positionalcomponent utilizes video feed from the image capture device 50 toprovide information about the position of an end effector, but otherinformation may be used instead of, or in addition to, this visualinformation, including sensor information, kinematic information, anycombination of these, or additional information that may provide theposition and/or orientation of the end effectors 66, 68, 70. Examples ofsystems that may be used for the tool tracking component 90 aredisclosed in, U.S. Pat. No. 5,950,629 (filed Apr. 28, 1994), U.S. Pat.No. 6,468,265 (filed Nov. 9, 1999), U.S. Pat. App. Pub. No. US2006/0258938 A1 (filed May 16, 2005), and U.S. Pat. App. Pub. No. US2008/0004603 A1 (filed Jun. 29, 2006). In accordance with an embodiment,the tool tracking component 90 utilizes the systems and methodsdescribed in commonly owned U.S. Pat. App. No. 61/204,084 (filed Dec.31, 2008). In general, the positional component maintains informationabout the actual position and orientation of end effectors. Thisinformation is updated depending upon when the information is available,and may be, for example, asynchronous information.

The kinematic component 92 is generally any device that estimates aposition, herein a “kinematic position,” of an end effector utilizinginformation available through the telesurgical system 20. In anembodiment, the kinematic component 92 utilizes kinematic positioninformation from joint states of a linkage to the end effector. Forexample, the kinematic component 92 may utilize the master/slavearchitecture for the telesurgical system 20 to calculate intendedCartesian positions of the end effectors 66, 68, 70 based upon encodersignals for the joints in the linkage for each of the tools 60, 62, 64.As examples, the kinematic component may utilize slave encoders 102and/or master manipulator encoders to estimate the position of tool. Anexample of system utilizing an embodiment of a kinematic component isdescribed in U.S. Pat. No. 7,155,315, which is incorporated herein byreference, although others may be utilized. Kinematic positioninformation for the end effector or any portion of the linkage and/ortool may also be provided in other ways, such as the use of opticalfiber shape sensing, sensing the positions of components (e.g.,electromagnetic components) embedded at various places along thelinkage, tool, or end effector, various video tool tracking methods,etc.

In the embodiment shown in the drawings, an error correction component94 is provided. In general, the error correction component calculates adifference between a location and/or orientation of a tool as providedby the tool tracking component 90 compared to the location and/ororientation of the tool as provided by the kinematic component 92.Because of the large number of joints and movable parts, currentkinematics measurement typically does not provide exact information forthe location of a surgical end effector in space. A system withsufficient rigidity and sensing could theoretically provide near-exactkinetic information. In current minimally invasive robotic surgerysystems, however, often the kinematic information may be inaccurate byup to an inch in any direction when taken in space. Thus, in accordancewith an embodiment, an offset may be generated by the error correctioncomponent 94. This offset provides information regarding the differencebetween the kinematic information provided by the kinematic componentand the actual position information provided by the tool trackingcomponent. Utilizing the offset, the kinematic information and theactual position information may be registered to the same locationand/or orientation.

In accordance with an embodiment, a modeling component 108 is providedfor generating a synthetic image 120 (FIG. 6) of a patient side cart,such as the patient side cart 40, or any portion thereof. In theembodiment shown in the drawings, the synthetic image 120 is of adifferent patient side cart configuration than the patient side cart 40(an illustrative model of a da Vinci® Surgical System Model IS2000patient side cart with three arms is shown), but the basic components ofthe two patient side carts are the same, except that the patient sidecart 40 includes an additional robotic arm assembly and tool. Inaccordance with an embodiment, the synthetic image 120 may be displayedon the display 84 or the viewer 32. To this end, modeling data 104 (FIG.3) may be provided that is associated with the vision cart 80 and/or thecomputer 82. The modeling data 104 may be, for example, atwo-dimensional (2-D) or three-dimensional (3-D) representation, such asan image, of the patient side cart 40, or any portion thereof. In anembodiment, such a representation is a 3-D model of the patient sidecart 40, or any portion thereof, and thus may represent an actual solidmodel of the patient side cart 40, or any portion thereof. The modelingdata 104 may be, for example, CAD data or other 3-D solid model datarepresenting components of the patient side cart 40. In an embodiment,the 3-D model is manipulatable at each joint of the patient side cart40, so that movements of the patient side cart may be mimicked by thesynthetic image 120 of the patient side cart 40. The modeling data mayrepresent the entire patient side cart or any portion thereof, such asonly the tools for the patient side cart.

Joint locations and orientations are generally known from kinematic dataprovided, for example, by the kinematic component 92. Utilizing thisinformation, each component of the patient side cart may be rendered inlocation so as to generate a image of the patient side cart that appearsin 3-D to the surgeon. Thus, in an embodiment, the modeling data 104includes individualized information for each component or link of thepatient side cart robot.

In accordance with an embodiment, the modeling component 108 constantlyupdates the location and/or orientation of the components of thesynthetic image 120 in accordance with information provided by the tooltracking component 90 and/or the kinematic component 92. For example, aninitial state of the kinematic component 92 may be determined includinga position of one or more end effectors for the patient side cart. Thesepositions may be compared with position information provided by the tooltracking component 90. As described above, the difference between theactual position as determined by the tool tracking component 90 and theestimated position of the end effectors provided by the kinematiccomponent 92 may result in an offset, which may be stored in orotherwise used by the error correction component 94. This offset may beused to register the position and orientation of an end effector asdetermined by the tool tracking component 90 to the position andorientation as estimated by the kinematic component 92.

As data is available from the tool tracking component 90, the actualposition of the end effector may be tracked and registered withinformation provided by the kinematic component 92. When tool trackinginformation is not available from the tool tracking component 90, anassumption may be made that any change in kinematic information providedby the kinematic component 92 is an indication of actual movement by theend effector. That is, when tool tracking is not available, the positionof an end effector may be accurately determined by the change incoordinate positions between the current position and the last knownposition, as calculated by the kinematic component 92. The assumptionhere is that the change in position may be accurately calculated usingonly kinematic data, without tool tracking information. This assumptionis reasonable, because although kinematic information is often notaccurate for calculating a position of an end effector in space, it istypically accurate for calculating a change of position once a positionis known, especially over a short period of time or for a small amountof movement. Thus, asynchronous data may be provided by the tooltracking component 90, and synchronous data may be provided by thekinematic component 92. The combination of this information providesdata regarding the positions and orientations of the components of thepatient side cart 40.

The positions of the components of a robotic arm assembly may bedetermined by utilizing the joint states provided by the kinematiccomponent. These joint states are calculated backwards from the endeffector, the position of which is known, as described above. Inaddition, because the slave encoders 102 at the joints of robotic armassemblies 122 for the patient side cart provide change in stateinformation for each joint, the relative position of each section of therobotic arm assemblies may be accurately estimated and tracked. Thus,information can be provided to the modeling component 108 that issufficient so that modeling component 108 may generate the syntheticimage 120 by utilizing the modeling data 104, with the position of eachof the segments of the robotic arm assemblies 122, including tools 124at the end of the robotic arm assemblies, or an endoscope 126 at the endof one of the robotic arm assemblies.

Referring again to FIG. 6, in an embodiment, in addition to thesynthetic image 120 for the patient side cart, a view volume 130 for theendoscope is provided. The view volume 130 represents a projection ofthe field of view of the endoscope 126. The field of view is the viewvisible by the endoscope, and the view volume is a projection of theboundaries of the field of view. That is, the view volume 130 representsa 3-D space that is visible by the endoscope 126. If desired, as shownin FIG. 4, camera information 132 may be provided to the modelingcomponent 108. The camera information includes a calibrated set ofintrinsic and extrinsic parameters about the camera. The intrinsicparameters include, e.g., focal length and principle point, which modelthe perspective mapping of the optics. Additionally, the intrinsicparameters may account for lens distortion. The extrinsic parameters mayaccount for, e.g., relative position and orientation between the stereoendoscopic views. As can be understood, changing the parameters, such aszoom, of the endoscope will change the view volume for the endoscope,such as making the view volume narrower or wider. In addition, as theendoscope 126 is moved, the view volume 130 will move accordingly. Thecamera information permits the creation of a 3-D stereo rendering thatmay be superimposed on the stereo view of the end effector from theimage capture device, as described below.

FIG. 7 is a flowchart representing a process for updating a rendering ofa synthetic image 120 in accordance with an embodiment. Beginning at401, the position and orientation of the patient side cart, or anyportion thereof, is sensed. This sensing may occur, for example, via thetool tracking component 90 and/or the kinematic component 92, asdescribed above.

At 402, the position and orientation information from 401 is used togenerate a model (e.g., the synthetic image 120). As described above,the modeling component 108 uses the modeling data 104 to generate themodel. The position and orientation information provided from 401 isutilized to correctly arrange the position and orientation of thesynthetic model to match that of the patient side cart.

At 404, as a result of the patient side cart moving, information isreceived. The movement may be, for example, movement of one of therobotic arm assemblies, movement of the endoscope, change in the focusof the endoscope, or movement by one of the end effectors. The movementof the end effector may be a change in location or orientation,including, for example, closing of pinchers or other operationalmovement of the end effectors.

At 406, a determination is made whether tool tracking information isavailable. In the embodiment show in FIG. 4, the determination iswhether an image is available so that the actual position of the endeffector or any portion of the tool that is in a field of view (e.g.,the view volume 130) of the endoscope 126 may be found using the tooltracking component 90. In one aspect, if tool tracking is available,then 406 branches to 408 where the tool tracking information is utilizedto update information about the position and orientation of the tooland/or end effector.

At 410, the kinematic information is used to update information aboutthe location and orientation of the joints of each linkage of the robotfor the patient side cart. At 412, the offset is updated, if desired. At414, the display of the synthetic image 120 is updated, and the processbranches back to 404.

At 406, if the tool tracking information is not available, then theprocess branches to 416, where the kinematic information provided by thekinematic component 92 is utilized to determine the position of the endeffector. The process then proceeds to 410, and then on through theprocess, although since the tool tracking information was not availableon this loop, the offset will likely not be updated, skipping 412.

Utilizing the method shown in FIG. 7, a 3-D rendering of the syntheticimage 120 is generated, and the synthetic image accurately representsthe physical configuration of the patient side cart at any point in timethroughout a surgical procedure. This information can be utilized andviewed by the surgeon S, or by someone else, to evaluate the state ofthe patient side cart. As described below, the viewer 34 or the display82 may show the synthetic image 120, either from a point of view that isthe same as the point of view from the endoscope, or from another angleor distance. The synthetic image 120 enables observation of all parts ofthe patient view cart via the viewer 32, thus permitting the surgeon Sto monitor movements of the robot and tools. In addition, in accordancewith an embodiment, viewing of these components is available inconnection with the view volume 130, permitting a surgeon to have a goodperspective of where the endoscope's field of view is with respect tospace. The view volume 130 provides a three dimensional representationof what is being seen by the surgeon S when looking in the viewer 32.

If desired, a single display may be provided for showing both the fieldof view of the endoscope and the synthetic image 120. For example, asshown in FIG. 8, a view 200 provided by the viewer 32 or the display 84provides both an actual field of view image 202 for the endoscope 126and the synthetic image 120. The synthetic image 120 is shown in aseparate tile window 204. In the embodiment shown in FIG. 8, the tile204 is approximately the same size as the field of view 202, but ifdesired, the tile window may be smaller or larger than the field of view202. Also, if desired, a toggle or other feature may be provided so thatthe surgeon may switch back and forth between a larger presentation ofthe synthetic image 120 or the field of view 202. In addition, thesynthetic image 120 and/or the tile window 204 may be partiallysuperimposed over a portion of the field of view, either on a continuousbasis or upon request.

As an example of toggling back and forth between a larger presentationof the synthetic image 120 or the field of view 202, a camera controlmay be provided that is connected to the master manipulators. Forexample, a user may start looking at the endoscopic view and may pullthe endoscope back by pulling the his hands towards himself while in acamera control mode. At some point, the endoscope cannot be pulled backany farther, and the field of view encompasses a maximum area.Continuing to pull back on the master controls (with or without a hapticdetent or other indication) can expose a view showing sections of asynthetic image 120 along the borders of the real image (e.g., the imagecaptured in field of view 202). Pulling back even farther on the mastercontrols (with or without haptic detent or other indication) may providea view where the image captured in field of view 202 is only the middlesection of the screen. Pulling back still farther on the controls (withor without haptic detent or other indication) may provide the entiresynthetic image 120. Reversing the master control direction can be usedto reverse such a real-to-synthetic zoom out function and control asynthetic-to-real zoom in function. As an alternative to camera controlusing master manipulator movement, the system may be configured to useanother control input (e.g., a foot pedal, a finger button on amanipulator, the roll of the master manipulator grip, and the like) tocontrol the zoom functions.

FIG. 9 shows a tile window 208 displaying an alternate angle for viewinga portion of the synthetic image 120. In the embodiment shown, the viewvolume 130 is slightly tilted from the actual field of view of theendoscope, but the particular angle of view of the view volume 130 showsrelevant information regarding the configuration of the tools 124 withrespect to the view volume.

The features of the synthetic image 120 provide another number ofbenefits to a user of the minimally invasive telesurgical system 20.Some of these advantages are set forth below.

Collision Detection

Typically, in a minimally invasive telesurgical system, only the mostdistal portions of the surgical tools, such as the tools 124, may bevisible to the surgeon in the field of view of the endoscope 126 at anytime. Depending upon the configuration of the patient side cart, it ispossible that collisions between moving parts of the robot assembly mayoccur which are not visible to the surgeon in the field of view. Some ofthese collisions (“outer collisions” because they are outside of thefield of view for the endoscope 126) may occur between the linkages ofrobotic arm assemblies leading to the tools, the collisions may occurbetween two tools, or may occur between a tool and a linkage. Such outercollisions may occur outside the body or inside the body but not withinthe field of view. In addition, an outer collision may occur between onetool that is in the field of view and another tool that is slightlyoutside the field of view. Collisions occurring inside the body and inthe field of view of the endoscope are “inner collisions”.

In accordance with an embodiment, the synthetic image 120 and/or theinformation generated by the modeling component 128 may be utilized forcollision detection. As an example, a surgeon viewing the viewer 32, oranother individual viewing the display 84, may view the synthetic image120 to see an indication of an imminent or actual collision.

Collision detection may involve more than just a visual image of acollision. Information about relative locations of robot linkages andtools is maintained by the modeling component 128, and this informationmay be used to generate a signal if two components are sensed to be tooclose to one another. For example, each tool may be treated like acapsule or cylinder, having a particular radius or buffer zone outsidethe tool's surface. Using the actual position information from the tooltracking component and/or the kinematic information from the kinematiccomponent 92, the modeling component 108 may predict or warn of acollision. For example, if two tools 124 are presumed to have a radiusof one half inch each, then if the center line for one of the toolscomes within an inch of the center line for a second tool, then themodeling component 108 may assume that a collision has occurred. Aseparate signal may be generated if the two tools are calculated to beclose, but not in contact, with each other. For the above example, thisdistance may be, e.g., a center line distance between the tools of 1.20inches.

FIG. 10 shows at the bottom a display tile window in which a real fieldof view image 250 shows two tools 252, 254 colliding. Although thecollision in FIG. 10 is within the field of view 250, as describedabove, the collision may take place outside the field of view or evenoutside the body of the patient. Even if inside the field of view, thetools 252, 254 are not necessarily visible, because they may be blockedby cauterization smoke, blood, or an organ, as examples. In FIG. 10, theinner collision is seen in the field of view 250, but it is alsodetected by the modeling component 108.

At the top of FIG. 10 is a display tile window 260 representing thesynthetic image 120. In the embodiment shown in FIG. 10, the tile window260 is taken from the same point of view as the field of view 250, but adifferent point of view may be provided as described above. In addition,as described above, outer collisions, as well as inner collisions, maybe detected.

FIG. 11 is a flowchart showing an illustrative process for providingcollision information in accordance with an embodiment. The processbegins at 1100. At 1102, a model, such as the synthetic image 120, isgenerated. This generation process is described with reference to FIG.7. At 1104, the robot for the patient side cart is moved. At 1105, theproximity of linkages and/or tools of the robotic arm assemblies 122 arecomputed. At 1106, a determination is made whether the proximities arewithin a high threshold. The high threshold represents spacing betweentools or linkages at which a warning of a collision is given. Forexample, as described above, if two tools are assumed to have a radiusof a half an inch, the high threshold may be a centerline separation of1.2 inches. If the components of the patient side cart are not withinthe high threshold, 1106 branches back to 1104, and the robot continuesto move.

If two components of the patient side cart are within the highthreshold, then 1106 branches to 1108, where a warning is generated.This warning may be an audible warning, a visual warning (e.g., providedwithin the viewer 32 or on the display 84), or another suitableindication of collision proximity. If visual, the warning may bepresented, for example, in the field of view 250 (FIG. 10). In theembodiment shown in FIG. 10, the words “inner collision error” areshown, indicating an actual collision. Alternatively, for a warningmessage, a message stating that tools are too close or similar may beprovided. In addition, for the view of the synthetic image 120, thecolor of the tools 124 may change to provide the warning, such aschanging from a metal color to yellow for a warning.

A surgeon may or may not elect to rearrange the robot after the warningis generated at 1108. In either event, the process proceeds to 1110,where the robot has moved again. At 1112, a determination is madewhether the robot is within a low threshold. In an embodiment, the lowthreshold represents a distance, such as a center line distance, atwhich a collision is assumed. If the low threshold is not met, theprocess branches back to 1104 and continues to loop, likely continuingto generate the warning message unless the components of the patientside cart are moved to outside the high threshold in 1106.

If the components are within the low threshold, then 1112 branches to1114, where collision information is generated, such as a collisionwarning or message. As an example, in FIG. 10, the collision errorwarning is provided in the field of view 250. (Both near and actualcollision warnings may use the same or different indications.) A similarcollision error warning may be provided in the tile window 260, and thetools 124 may change colors, such as to red, to show a collision error.The process then loops back to 1104.

As stated above, for collision detection, the components need not be inthe field of view of the viewer 32. Thus, when components of the patientside cart are improperly aligned and are approaching a collision oractually have a collision, information may be provided, either in visualform or in the form of a warning or error message. The warning may beparticularly helpful where a user is not familiar with operation of therobot and may put the tools or robotic arm assemblies in an awkwardposition. The person viewing the viewer 32 may select a differentsynthetic view angle and distance of the robot so as to determine thenear collision or actual collision point between two roboticmanipulators. Once the operator views the collision point, he or she mayadjust one or more of the robot's kinematic arms (either the passive,“set up” portions or the actively controlled, manipulator portions) tocure the actual or near collision condition and avoid furthercollisions. In one aspect, if the operator is viewing a synthetic viewthat corresponds to the endoscope's field of view, the synthetic viewmay be automatically changed to show a collision point if a collisionwarning or actual collision is occurring.

In an embodiment, the location of a patient and/or portions of thepatient's tissue structures (e.g., from preoperative imaging or by othersuitable method of registering tissue structure locations) may beprovided to the system, and registered patient location data may be todetect, warn, and display actual or potential collisions between therobot and the patient or designated tissue structures in the patient.Collisions may be detected as described above.

Also, in an embodiment, a visual, audio, or other indicator may beprovided to assist in reducing or correcting a collision state. Forexample, for the warning situation described above, information may beprovided to a surgeon to aid the surgeon in avoiding a collision. Forexample, a visual indicator may provide information about a movementdirection in which a collision might occur, or may indicate a movementdirection for the surgeon to make in order to avoid or cure a collision.

Lost Tool Recovery

In minimally invasive surgery, it is possible for instruments to bepositioned outside the endoscopic camera's view volume. This possibilitycan result in situations where the tool is effectively lost, since thesurgeon does not necessarily know how to move the endoscope to bring theinstrument back into view, or how to move the instrument into theendoscope's field of view. Moreover, the situation may compromisepatient safety, since the surgeon is able to move an instrument whichcannot be observed.

The synthetic image 120 provides a solution to this problem bypresenting the surgeon with a broader view of the endoscope's viewvolume 130, along with an accurate depiction of the position of eachtool 124. Such a broader view and tool depiction may be provided fromvarious points of view. In an embodiment, the broad view and tooldepictions are provided from the same point of view or direction as theendoscope field of view. By providing a broad view in this direction,the surgeon will be able to retain the intuitive tool control movementhe or she normally experiences when viewing the real endoscopic imagewhile moving tools into the proper position so that the tool is back inthe view volume 130. Alternatively, the view volume 130 may be viewedfrom other angles, allowing a surgeon to have a different perspective ofwhat the endoscope 126 is viewing. As examples, FIGS. 8 and 9 show threedifferent views, taken at different angles and pans, of views that maybe shown for the synthetic image 120. Although the lower part of FIG. 8shows an actual image, a synthetic image 120 may be provided from thesame direction, and would look similar except that synthetic tools wouldbe shown instead of video feed of the actual tools. The view establishedby the field of view is shown in the lower part of FIG. 8, and a viewtaken from a front side of the synthetic image—zoomed outward to showmuch of the patient side cart—is shown in the top of FIG. 8. A viewtaken slightly rearward and upward of the direction of the field of viewof the endoscope, and zoomed outward to show the view volume 130, isshown in FIG. 9. This slight variation in view provides a goodperspective of where the tools 124 are with respect to the view volume130. A surgeon may toggle between a view consistent with the field ofview and one just off from the field of view, such as shown in FIG. 9.To this end, a controller or other device may be provided for allowing asurgeon to toggle between different views of the synthetic image 120.Alternatively, a separate controller or the master controller may beutilized to allow infinite positioning (e.g., various pan, tilt, roll,dolly, truck, crane, and zoom image movements) of the synthetic image120.

FIG. 12 is a flow chart representing a process for lost tool recovery inaccordance with an embodiment. The process begins at 1200. At 1202, thesynthetic image 120 is generated as described above. At 1204, thepatient side cart, or the robot, is moved.

At 1206, a determination is made whether one or more of the tools isoutside of the field of view. If not, the process loops back to 1204. Ifone or more of the tools is outside of the field of view, then theprocess may move to 1208, where a synthetic image is shown. Thesynthetic image may or may not be automatically shown; the syntheticimage display may be selected by a surgeon. To this end, 1208 may bedone as a result of a request by the surgeon or another operator, andmay or may not be triggered by a tool being out of the field of view. Ifdesired, however, a synthetic image may be automatically shown as aresult of a loss of an image of the tool. In such an embodiment,however, it may be desirable to show the synthetic image in a tilewindow in addition to the field of view, instead of taking the field ofview away from the surgeon.

If the missing tool display option is available, the synthetic view 120may be requested or otherwise provided in 1208. The synthetic imageprovided in 1208 may be, as described above, substantially the same asthe field of view of the endoscope 126 or any number of perspectives ofthe modeled system. If a desired angle is not shown, then a surgeon mayelect at 1210 to show a different view. If the surgeon elects to show adifferent view, then 1210 branches to 1212, where the synthetic image120 is, e.g., rotated to show a different view. If desired, as part ofthis movement, the synthetic image may rotate in space so that thesurgeon may get an idea of the position from which the view startedrelative to the position where the view is going. In addition, inaccordance with an embodiment, when a view of the synthetic image 120 isinconsistent with the same point of view as the field of view, a warningmessage or other indicator may be provided to the surgeon so that thesurgeon may understand that he or she is looking at the view volume 130from a direction that is different than the direction of the field ofview.

If the surgeon did not request a different view in 1210, then theprocess loops back to 1204.

As described above, the synthetic image 120 provides an image of thepatient side cart that is larger than and outside of the view volume130. Thus, even if taken along the same point of view as the field ofthe view of the endoscope 126, the surgeon may zoom outward so thattools that are just outside the view volume 130 may be seen. The surgeonmay then move these tools or the endoscope to the desired position sothat they are within the field of view.

Mixed Video and Rendered View

As described above, there are a number of ways in which the system maypresent the synthetic image 120 of the robot to the surgeon. A firstoption, described with respect to FIG. 8, includes a tile window 204showing a synthetic view above the field of view image 202, with bothshown at the same time. Another option, shown in FIG. 9, shows only thesynthetic image 120.

In accordance with an embodiment, a third option is provided in which avideo display from an endoscope is superimposed over the synthetic image120, with the positions matched, so that the video image is rendered inthe context of the synthetic image 120 of the entire patient side cart.This view provides relative positions of the components of the patientcart for the surgeon, and allows the surgeon to understand where thesurgeon is with respect to space. The view is also well suited whentransitioning between a pure video display and a pure synthetic image120. During the transition, the surgeon can relate respective positionsof the robot and the video image from the endoscope.

A simplified version of this feature is shown in FIG. 13, where an imagewithin the field of view 300 is projected over a window tile 306 thatincludes the synthetic image 120. The field of view image 300 includestwo tools 302, 304 performing an operation. The window tile 306 extendsthe view provided by the field of view 300, and additional sections ofthe tools 302,304—indicated by the reference numerals 308,310,respectively—are provided. The surgeon may zoom in and out to provideadditional information about the location of the tools with respect toother parts of the patient side cart. In addition, the featuresdescribed with respect to the embodiment shown in FIG. 13 may beutilized to find the lost tool that is just outside the field of view,for example, in the window tile 306, but not in the field of view 300.

Visual Troubleshooting Indicator

In accordance with an embodiment, instead of or in addition to thesynthetic image 120, the modeling data 104 may be utilized to project aimage other than a visual representation of portions of the patient sidecart. For example, using the position information provided by the tooltracking component 90 and/or the kinematic component 92, the modelingcomponent 108 may display a portion of the synthetic image 120 in adifferent color, or it may display text on a portion of the syntheticimage or instead of the synthetic image. In such an embodiment, the textmay be superimposed over the actual tools in a field of view so as tofocus attention on that tool or to provide other information. As anexample, for the tool 304 in FIG. 13, the modeling component 108 may beutilized to display a text message “closed” 320 collocated over thevideo image of the tool 304 to indicate that the clamp for the tool isclosed. The camera information, described above, permits the creation ofa 3-D stereo rendering that may be superimposed on the stereo view ofthe tool 304 from the image capture device. Error messages may also beprovided.

FIG. 14 is a flow chart representing a process for displayinginformation utilizing the modeling component 108 in accordance with anembodiment. Beginning at 1400, the location of the components of thepatient side cart is determined, for example, the location of the tools124. At 1402, the modeling component 108 is aligned with the tool asdescribed above. At 1404, the desired information is displayed over thetool. For example, as described above, words may be displayed over thetool. In addition, if desired, information may be displayed around oradjacent to a tool or other feature.

As can be understood, to superimpose a message over actual tools in thefield of view, the modeling data 104 need only include information aboutthe outer perimeter of the tools. The other components of the patientside cart are not needed for this embodiment.

Communication Aid

The synthetic image 120 may be useful in providing a remote image of theoperation of the patient side cart. For example, in some situations, anindividual remote from the patient side cart may desire to viewoperation of the patient side cart. In such a situation, the syntheticimage 120 may be rendered at both the viewer 32 and a remote display(e.g., the display 84). In such a situation, in accordance with oneembodiment, the modeling data may be maintained all at one location,with the synthetic image 120 sent to a remote location for display atthe remote location.

In an alternate embodiment, position and orientation informationprovided by the tool tracking component 90 and/or the kinematiccomponent 92 may be sent to a remote computer. The remote computer, inturn, includes a modeling component 108 and the modeling data 104. Inthis embodiment, the synthetic image 120 is generated at the remotelocation in a separate operation from producing the synthetic image 120for the viewer 32.

Being able to provide a synthetic image 120 in remote locations permitsan operating surgeon viewing the surgeon's console to communicate with asurgical assistant viewing an assistant monitor. In addition, a studentsurgeon at one surgeon console may communicate with a remote proctor atanother surgeon console.

In accordance with another embodiment, a remote user or proctor may havecontrols for movement of a synthetic image, such as a synthetic image120. The movement of the synthetic image may be watched by a surgeon orstudent at the surgeon console, permitting the user to learn surgicalprocedures and motions, and to mimic those motions with the surgeon orstudent's controls (and thus the tools).

Range of Motion Limits

The linkages for the robotic arm assemblies of the patient side carthave a limited range of movement, limiting the movement of the toolssupported by each arm or linkage. When the robot for a patientencounters range of motion limits, it is not always obvious to a surgeon(new or experienced) why the robot is not able to continue moving. In atelesurgical system, there are typically two sources of range of motionlimits: joint limits of the master manipulator and joint limits of theslave manipulator.

In accordance with an embodiment, the modeling component 108 generates asignal to indicate that a limit of the range of movement for a tool isapproaching. The signal may be used, for example, to generate a visualcue to the surgeon, such as color coding of the part(s) that havereached a limit. Alternatively, the limit may be represented withsynthetic geometry as a virtual wall 340 (FIG. 6), which may be shownwith the synthetic model 120, or may alternately be superimposed overthe field of view. The virtual wall 340 is for the right-most tool 124,and it may be shown as concave, flat, or otherwise shaped to match thecurvature of a range of motion. The virtual wall 340 is displayed in aposition and direction that is perpendicular to the impeded motiondirection of the instrument tip.

Other variations are within the spirit of the present invention. Thus,while the invention is susceptible to various modifications andalternative constructions, a certain illustrated embodiment thereof isshown in the drawings and has been described above in detail. It shouldbe understood, however, that there is no intention to limit theinvention to the specific form or forms disclosed, but on the contrary,the intention is to cover all modifications, alternative constructions,and equivalents falling within the spirit and scope of the invention, asdefined in the appended claims.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the invention (especially in the context of thefollowing claims) are to be construed to cover both the singular and theplural, unless otherwise indicated herein or clearly contradicted bycontext. The terms “comprising,” “having,” “including,” and “containing”are to be construed as open-ended terms (i.e., meaning “including, butnot limited to,”) unless otherwise noted. The term “connected” is to beconstrued as partly or wholly contained within, attached to, or joinedtogether, even if there is something intervening. Recitation of rangesof values herein are merely intended to serve as a shorthand method ofreferring individually to each separate value falling within the range,unless otherwise indicated herein, and each separate value isincorporated into the specification as if it were individually recitedherein. All methods described herein can be performed in any suitableorder unless otherwise indicated herein or otherwise clearlycontradicted by context. The use of any and all examples, or exemplarylanguage (e.g., “such as”) provided herein, is intended merely to betterilluminate embodiments of the invention and does not pose a limitationon the scope of the invention unless otherwise claimed. No language inthe specification should be construed as indicating any non-claimedelement as essential to the practice of the invention.

Preferred embodiments of this invention are described herein, includingthe best mode known to the inventors for carrying out the invention.Variations of those preferred embodiments may become apparent to thoseof ordinary skill in the art upon reading the foregoing description. Theinventors expect skilled artisans to employ such variations asappropriate, and the inventors intend for the invention to be practicedotherwise than as specifically described herein. Accordingly, thisinvention includes all modifications and equivalents of the subjectmatter recited in the claims appended hereto as permitted by applicablelaw. Moreover, any combination of the above-described elements in allpossible variations thereof is encompassed by the invention unlessotherwise indicated herein or otherwise clearly contradicted by context.

1-38. (canceled)
 39. A system comprising: a first robotic arm adapted tosupport and move a tool; a second robotic arm adapted to support andmove a camera; an input device; a display; and a processor configuredto: in a first mode, command the first robotic arm to move the camera inresponse to a first input received from the input device to capture animage of the tool, and present the image as a displayed image on thedisplay; and in a second mode, display a synthetic image of the firstrobotic arm in a boundary area around the captured image on the display,and in response to a second input, change a size of the boundary arearelative a size of the displayed image.
 40. The system of claim 39,wherein the second input is received at the input device.
 41. The systemof claim 40, wherein the first input comprises an interaction with theinput device, and wherein the second input comprises the interactionwith the input device.
 42. The system of claim 41, wherein the image isof a surgical site, wherein in the first mode the interaction causes thecamera to move away from the surgical site, and wherein in the secondmode the interaction causes the size of the boundary area to increaseand causes the size of the displayed image to decrease.
 43. The systemof claim 42, wherein the synthetic image is generated via a syntheticimaging component, and wherein in the second mode the interactionincreases a field of view of the synthetic imaging component.
 44. Thesystem of claim 39, wherein the second input is received at a secondinput device.
 45. The system of claim 44, wherein the processor isfurther configured to, in a third mode, zoom the image captured by thecamera in response to a third input at the second input device.
 46. Thesystem of claim 45, wherein the second input comprises an interactionwith the second input device, and the third input comprises theinteraction with the second input device.